On-camera image processing based on image luminance data

ABSTRACT

A camera system processes images based on image luminance data. The camera system includes an image sensor, an image pipeline, an encoder and a memory. The image sensor converts light incident upon the image sensor into raw image data. The image pipeline converts raw image data into color-space image data and calculates luminance levels of the color-space image data. The encoder can determine one or more of quantization levels, determining GOP structure or reference frame spacing for the color-space image data based on the luminance levels. The memory stores the color-space image data and the luminance levels.

CROSS REFERENCE TO RELATED APPLICATIONS

This application claims the benefit of U.S. Provisional Application No.62/339,405 of the same title, filed May 20, 2016, and U.S. ProvisionalApplication No. 62/339,402 entitled “ON-CAMERA IMAGE PROCESSING BASED ONIMAGE ACTIVITY DATA”, filed May 20, 2016, each of which is incorporatedby reference in its entirety.

BACKGROUND

Field of Art

The present application is related to commonly owned U.S. patentapplication Ser. No. 15/593,630, entitled “ON-CAMERA IMAGE PROCESSINGBASED ON IMAGE ACTIVITY DATA” AND FILED May 12, 2017.

The disclosure generally relates to the field of digital image and videoprocessing, and more particularly to image luminance data and imageactivity data within a camera architecture.

Description of the Related Art

As encoding technology improves, in-camera encoders are better able toencode images and videos in real time. However, real-time encoders oftensuffer from lossy content encoding issues, such as losing image qualityand data associated with luminance and/or activity variance. In digitalcamera systems, such encoder issues can hinder camera capabilities.Furthermore, even if an encoder has the capability to encode images andvideos without losing too much detail, the encoder may use largerbandwidth with larger latency and higher power than a typical camera canprovide.

Summary

In one aspect of the present disclosure, a camera system is disclosed.In one embodiment, the camera system includes: an image sensorconfigured to convert light incident upon the image sensor into rawimage data; an image pipeline configured to: convert the raw image datainto color-space image data; and calculate luminance levels of thecolor-space image data; an encoder configured to: determine quantizationlevels of the color-space image data based on the luminance levels; andencode the color-space image data using the determined quantizationlevels to produce encoded image data; and a memory configured to storethe encoded image data.

In another aspect of the present disclosure, a method is disclosed. Inone embodiment, the method includes: capturing, by a camera, lightincident upon an image sensor to produce raw image data; converting, bythe camera, the raw image data into color-space image data; calculating,by the camera, luminance levels of the color-space image data;determining, by the camera, quantization levels of the color-space imagedata based on the luminance levels; and encoding, by the camera, thecolor-space image data using the determined quantization levels toproduce encoded image data.

BRIEF DESCRIPTION OF DRAWINGS

The disclosed embodiments have other advantages and features which willbe more readily apparent from the detailed description, the appendedclaims, and the accompanying figures (or drawings). A brief introductionof the figures is below.

FIG. 1 illustrates an example high-level block diagram of a camerasystem for processing images based on image luminance data and/or imageactivity data within a digital image, according to one embodiment.

FIG. 2 illustrates an example high-level block diagram of a camerasystem for processing images based on image luminance data, according toone embodiment.

FIG. 3 illustrates an example high-level block diagram of a camerasystem for processing images based on image activity data, according toone embodiment.

FIG. 4A illustrates an image captured by an image sensor having portionswith different luminance levels and activity variances, according to oneembodiment.

FIG. 4B illustrates a diagram showing relationships between quantizationlevels and the luminance levels of image portions illustrated in FIG.4A, according to one embodiment.

FIG. 4C illustrates a diagram showing relationships between quantizationlevels and the activity variances of image portions illustrated in FIG.4A, according to one embodiment.

FIG. 5A illustrates a diagram of determining a reference frame of aseries of frames based on image activity data, in according to oneembodiment.

FIG. 5B illustrates a diagram of determining a block type based on imageactivity data, in according to one embodiment.

FIG. 5C illustrates a diagram of determining a transform size based onimage activity data, in according to one embodiment.

FIG. 6 illustrates an example block diagram of an image processingengine, according to one embodiment.

FIG. 7 illustrates a process for encoding image data based on imageluminance data, according to one embodiment.

FIG. 8 illustrates a process for encoding image data based on imageactivity data, according to one embodiment.

DETAILED DESCRIPTION

The figures and the following description relate to preferredembodiments by way of illustration only. It should be noted that fromthe following discussion, alternative embodiments of the structures andmethods disclosed herein will be readily recognized as viablealternatives that may be employed without departing from the principlesof what is claimed.

Reference will now be made in detail to several embodiments, examples ofwhich are illustrated in the accompanying figures. It is noted thatwherever practicable similar or like reference numbers may be used inthe figures and may indicate similar or like functionality. The figuresdepict embodiments of the disclosed system (or method) for purposes ofillustration only. One skilled in the art will readily recognize fromthe following description that alternative embodiments of the structuresand methods illustrated herein may be employed without departing fromthe principles described herein.

Example Camera Configuration

FIG. 1 illustrates an example high-level block diagram of a camerasystem for processing images based on image luminance data and/or imageactivity data within a digital image, according to one embodiment. Thecamera 100 of the embodiment of FIG. 1 includes one or moremicrocontrollers 102, a system memory 104, a synchronization interface106, a controller hub 108, one or more microphone controllers 110, oneor more image sensors 112, a lens and focus controller 114, an imageprocessing engine 116, one or more lenses 120, one or more LED lights122, one or more buttons 124, one or more microphones 126, an I/O portinterface 128, a display 130, and an expansion pack interface 132.

The camera 100 includes one or more microcontrollers 102 (such as aprocessor) that control the operation and functionality of the camera100. For instance, the microcontrollers 102 can execute computerinstructions stored on the memory 104 to perform the functionalitydescribed herein. In some embodiments, the camera 100 can capture imagedata, can provide the image data to an external system (such as acomputer, a mobile phone, or another camera), and the external systemcan generate a color map and a tone map based on the captured imagedata.

A lens and focus controller 114 is configured to control the operation,configuration, and focus of the camera lens 120, for instance based onuser input or based on analysis of captured image data. The image sensor112 is a device capable of electronically capturing light incident onthe image sensor 112 and converting the captured light to image data.The image sensor 112 can be a CMOS sensor, a CCD sensor, or any othersuitable type of image sensor, and can include correspondingtransistors, photodiodes, amplifiers, analog-to-digital converters, andpower supplies.

A system memory 104 is configured to store executable computerinstructions that, when executed by the microcontroller 102, perform thecamera functionalities described herein. The system memory 104 alsostores images captured using the lens 120 and image sensor 112. Thememory 104 can include volatile memory (e.g., random access memory(RAM)), non-volatile memory (e.g., a flash memory), or a combinationthereof. In some embodiments, the memory 104 may be a double data ratesynchronous dynamic random-access memory (DDR), an on-chip buffer, or anoff-chip buffer. The memory stores data processed by the imageprocessing engine 116.

A synchronization interface 106 is configured to communicatively couplethe camera 100 with external devices, such as a remote control, anothercamera (such as a slave camera or master camera), a computer, or asmartphone. The synchronization interface 106 may transfer informationthrough a network, which allows coupled devices, including the camera100, to exchange data other over local-area or wide-area networks. Thenetwork may contain a combination of wired or wireless technology andmake use of various connection standards and protocols, such as WiFi,IEEE 1394, Ethernet, 802.11, 4G, or Bluetooth.

A controller hub 108 transmits and receives information from user I/Ocomponents. In one embodiment, the controller hub 108 interfaces withthe LED lights 122, the display 130, and the buttons 124. However, thecontroller hub 108 can interface with any conventional user I/Ocomponent or components. For example, the controller hub 108 may sendinformation to other user I/O components, such as a speaker.

A microphone controller 110 receives and captures audio signals from oneor more microphones, such as microphone 126A and microphone 126B.Although the embodiment of FIG. 1 illustrates two microphones, inpractice, the camera can include any number of microphones. Themicrophone controller 110 is configured to control the operation of themicrophones 126. In some embodiments, the microphone controller 110selects which microphones from which audio data is captured. Forinstance, for a camera 100 with multiple microphone pairs, themicrophone controller 110 selects one microphone of the pair to captureaudio data.

The image processing engine 116 uses an image pipeline to calculateimage luminance data (also referred to as luminance levels) and/or imageactivity data (also referred to as activity variances) for the imagedata captured by the image sensor 112, and to convert the captured imagedata into a color-space image data. Examples of a color space mayinclude a RGB-type color space (e.g., sRGB, Adobe RGB, Adobe Wide GamutRGB, etc.), a CIE defined standard color space (e.g., CIE 1931 XYZ,CIELUV, CIELAB, CIEUVW, etc.), a Luma plus chroma/chrominance-basedcolor space (e.g., YIQ, YUV, YDbDr, YPbPr, YCbCr, xvYCC, LAB, etc.), ahue and saturation-based color space (e.g., HSV, HSL), or a CMYK-typecolor space. For example, the image processing engine 116 can convertRAW RGB image data into YUV image data. In some embodiments, the imageprocessing engine 116 identifies a plurality of blocks of the capturedimage data and calculates luminance level and/or activity variance foreach block (for instance, by determining an average luminance oractivity for all pixels of the captured image data or portions of eachblock of the captured image data (or, likewise, scaled up or scaled downpixels of the captured image data). The image processing engine 116stores the converted color-space (e.g., YUV data) image data andcorresponding luminance levels and/or activity variances into the memory104.

The image processing engine 116 includes an encoder to generatecompressed encoded data using the converted color-space image data andcorresponding luminance levels and/or activity variances extracted fromthe memory 104. Examples of an encoder may include an H.264 encoder, anH.265/HEVC encoder, a VP9 encoder or a JPEG encoder. In someembodiments, the image processing engine 116 selects a quantizationlevel, a block type such as Intra or Inter block, and/or a transformsize and/or type for each block of the converted color-space image databased on a corresponding luminance level and/or activity variance. Insome embodiments, the image processing engine 116 selects a frame type(such as Intra or Inter frame), or selects group of pictures (GOP)structure for encoding based on a corresponding luminance level and/oractivity variance. Examples of the image processing engine 116 arefurther described in detail below with regard to FIGS. 3-5.

In some embodiments, the image processing engine 116 may include apre-encoding module to further process image data before encoding withinthe image pipeline of the camera 100. For example, the image pipelinecan output converted color-space image data and corresponding luminancelevels and/or activity variances. The image processing engine 116 maypre-process this output using one or more pre-processing operations,such as operations reversing effects of geometric distortions caused bythe lens 120 (e.g., Dewarping), operations reducing noise (e.g., 3Dnoise reduction), and operations reducing blurring associated withmotions of the camera 100 during exposure (e.g., electronic imagestabilization). It should be noted that in some embodiments, suchpre-processing operations are performed on the image data beforeencoding, for instance before the encoded image data is stored by thecamera 100.

Additional components connected to the microcontroller 102 include anI/O port interface 128 and an expansion pack interface 132. The I/O portinterface 128 may facilitate the camera 100 in receiving or transmittingvideo or audio information through an I/O port. Examples of I/O ports orinterfaces include USB ports, HDMI ports, Ethernet ports, audioports,and the like. Furthermore, embodiments of the I/O port interface 128 mayinclude wireless ports that can accommodate wireless connections.Examples of wireless ports include Bluetooth, Wireless USB, Near FieldCommunication (NFC), and the like. The expansion pack interface 132 isconfigured to interface with camera add-ons and removable expansionpacks, such as an extra battery module, a wireless module, and the like.

Image Processing Based on Image Luminance Data and/or Image ActivityData

FIG. 2 illustrates an example high-level block diagram of a camerasystem for processing images based on image luminance data, according toone embodiment. In the embodiment of FIG. 2, the imaging processingengine 116A includes a luminance-based image pipeline 230 and an encoder220. In some embodiments, the imaging processing engine 116A may includedifferent and/or additional components than those illustrated in theembodiment of FIG. 2 to perform the functionalities described herein.

As shown in FIG. 2, the luminance-based image pipeline 230 receives thecaptured image data from the image sensor 112. The luminance-based imagepipeline 230 converts the captured image data into YUV data 215A andcalculates luminance levels (e.g., Y levels 213A) for the YUV data 215A.For example, the luminance-based image pipeline 230 identifies aplurality of blocks (e.g., 4×4 blocks) in the YUV data 215A. For eachblock, the luminance-based image pipeline 230 may use a sum or averageof pixel values in the block as a luminance level (Y level 213A). Inthis example, the luminance-based image pipeline 230 can determine a Ylevel for each block within the YUV data 215A. In some embodiments, a Ylevel is directly proportional to a brightness of a corresponding block,such that a Y level for a first block is greater than a Y level for asecond block that is darker than the first block. The luminance-basedimage pipeline 230 stores the YUV data 215A and the Y levels 213A intothe memory 104.

The encoder 220 extracts the YUV data 215B and the Y levels 213B fromthe memory 104. The encoder 220 selects a quantization level for eachblock of the YUV data 213B based on the Y level 213B corresponding tothat block. For example, if the Y level 213B for a block indicates thatthe block is below a threshold level of brightness, the encoder 220 canselect a below-threshold quantization level (e.g., a strength ofquantization that is below a threshold) for the block such that theblock, when compressed using the selected quantization level, iscompressed in such a way as to preserve an above-threshold amount ofimage information within the block. If the Y level 213B for a blockindicates that the block is above a threshold level of brightness, theencoder 220 can select an above-threshold quantization level (e.g., astrength of quantization that is above a threshold) for the block suchthat the block, when compressed using the selected quantization level,is compressed by an above-threshold compression factor. In someembodiments, if the Y level 213B for a block indicates that the block isabove a threshold level of brightness, the encoder 220 can select abelow-threshold quantization level for the block such that the block,when compressed using the selected quantization level, is compressed insuch a way as to preserve an above-threshold amount of image informationwithin the block. If the Y level 213B for a block indicates that theblock is below a threshold level of brightness, the encoder 220 canselect an above-threshold quantization level for the block such that theblock, when compressed using the selected quantization level, iscompressed by an above-threshold compression factor. The encoder 220generates compressed encoded data 225 by applying different quantizationlevels to different blocks of the YUV data 215B based on luminancelevels. The compressed encoded data 225 may include less-compresseddarker image portions with above-threshold amounts of image informationpreserved, and more-compressed brighter portions (or vice-versa). Assuch, the image processing engine 116A performs the image processingbased on image luminance data without significantly increasing systembandwidth or power. Additionally and/or alternatively, the encoder 220uses Y level 213B to determine a block type (such as Intra, Inter, orskip block), a transform size, a transform type, or spacing betweenreference frames (such as I frames) or the GOP structure of an encodedstream (such as a series of encoded images or frames). Examples arefurther described in detail below with regard to FIG. 5A through FIG.5C.

Typically, human eyes are more susceptible to notice flaws or artifactsin flat, smooth, or non-detailed areas of an image. For example, any“blocky”-type image artifacts in a portion of an image corresponding tothe sky will be very noticeable. Likewise, human eyes are lesssusceptible to find (or care about) details of dense areas (such as thetree in a moving picture). To overcome this problem, an image can beencoded based on image activity (such as the activity variance). Imageactivity describes details of objects that are captured in an image. Forexample, flat areas such as the sky and smooth walls have less activitywhile dense areas such as tree leaves and water from a water fountainhave high activity. Activity variance describes a difference in activityamong blocks of a same image (or frame), or among blocks of differentimages (or frames). For example, for a single image, the activityvariance of the image describes areas of the image as ‘dense’ or ‘flat’.The ‘dense’ or ‘flat’ areas can be encoded differently such that thequality of encoding is maintained uniformly across the image. Forexample, more dense areas may take more bits to encode (in order topreserve more details) while less dense areas may take fewer bits toencode. Examples are further described in detail below with regard toFIG. 4A through FIG. 4C. In another example, the activity variancedescribes a difference in activity between corresponding portions ofmultiple frames. Examples are further described in detail below withregard to FIG. 5A through FIG. 5B.

FIG. 3 illustrates an example high-level block diagram of a camerasystem for processing images based on image activity data, according toone embodiment. In the embodiment of FIG. 3, the imaging processingengine 116B includes an activity-based image pipeline 330, and anencoder 320. In some embodiments, the imaging processing engine 116B caninclude different and/or additional components than those illustrated inthe embodiment of FIG. 3 to perform the functionalities describedherein.

As shown in FIG. 3, the activity-based image pipeline 330 receives thecaptured image data from the image sensor 112. The activity-based imagepipeline 330 converts the captured image data into YUV data 315A andcalculates image activity data (e.g., activity variances ACT Vars. 313A)for the YUV data 315A. For example, the activity-based image pipeline330 identifies a plurality of blocks (e.g., 4×4 blocks) in the YUV data315A. For each block, the activity-based image pipeline 330 may add asum or average of differences of pixel values along a vertical directionof the YUV data 315A and a sum or average of differences of pixel valuesalong a horizontal direction of the YUV data 315A together as anactivity variance. In this example, the activity-based image pipeline330 can determine ACT Var. 313A for each bock within the YUV data 315A.In some embodiments, an ACT var. is directly proportional to amounts ofactivities of a corresponding block, such that an ACT Var. for a firstblock is greater than an ACT Var. for a second block that has largeramounts of activities than the first block. The activity-based imagepipeline 330 stores the YUV data 315A and the Y levels 313A into thememory 104.

The encoder 320 extracts the YUV data 315B and the ACT Vars. 313B fromthe memory 104. The encoder 320 determines a quantization level for eachblock of the YUV data 315B based on the ACT Vars. 313B corresponding tothat block. For example, if the ACT Var. 313B for a block indicates thatthe block is above a threshold level of activities, the encoder 320 canselect a below-threshold quantization level (e.g., a strength ofquantization that is below a threshold) for the block such that theblock, when compressed using the selected quantization level, iscompressed in such a way as to preserve an above-threshold amount ofimage information within the block. If the ACT Var. 313B for a blockindicates that the block is below a threshold level of activities, theencoder 320 can select an above-threshold quantization level (e.g., astrength of quantization that is above a threshold) for the block suchthat the block, when compressed using the selected quantization level,is compressed by an above-threshold compression factor.

In some embodiments, if the ACT Var. 313B for a block indicates that theblock is below a threshold level of activity, the encoder 320 can selecta below-threshold quantization level for the block such that the block,when compressed using the selected quantization level, is compressed insuch a way as to preserve an above-threshold amount of image informationwithin the block. If the ACT Var. 313B for a block indicates that theblock is above a threshold level of activities, the encoder 320 canselect an above-threshold quantization level for the block such that theblock, when compressed using the selected quantization level, iscompressed by an above-threshold compression factor. The encoder 320generates compressed encoded data 325 by applying different quantizationlevels to different blocks of the YUV data 315B based on activityvariances. The compressed encoded data 325 may include less-compressedhigher-activity image portions with above-threshold amounts of imageinformation preserved, and more-compressed lower-activity portions (orvice-versa). As such, the image processing engine 116A performs theimage processing based on image activity data without significantlyincreasing system bandwidth or power. Additionally and/or alternatively,the encoder 320 uses ACT Vars. 313B to determine a block type (such asIntra, Inter or skip block), transform size of the block, or type oftransform, or spacing between reference frames (such as I frames) or theGOP structure of an encoded stream (such as a series of encoded imagesor frames).

FIG. 4A illustrates an image 400 captured by the image sensor 112 havingportions with different luminance levels and activity variances,according to one embodiment. As shown in FIG. 4A, the image 400 may bedivided into four portions. Each portion may include one or more blocks.A first portion 410 includes a bonfire and has a brightest luminance anda highest activity. A second portion 420 includes a boy and has a darkluminance (e.g., less light on the face and body) and low activity(e.g., non-textured clothing and more flat areas). A third portion 430includes a girl and has a dark luminance (e.g., less light on face andbody) and high activity (e.g., details on marshmallows, texturedclothing, etc.). A fourth portion 440 includes trees and has a darkestluminance and high activity (e.g., dense leaves).

FIG. 4B illustrates a diagram 450 showing relationships betweenquantization levels 470 and the luminance levels of image portionsillustrated in FIG. 4A, according to one embodiment. The goal is topreserve image quality at low light while maintaining quality at wellilluminated areas in the image. As shown in FIG. 4A, a lowerquantization level is applied to the low luminance level areas and ahigher quantization level is applied to high luminance areas. Therelationship between luminance levels and quantization level among theportions is: the quantization level for the first portion 410 (with thehighest luminance level) is greater than the quantization level for thesecond portion 420 and the third portion 430 (with moderate luminancelevels), which are greater than the quantization level for the fourthportion 440 (with the lowest luminance level). Thus, the highestquantization level is applied to the first portion 410, the lowestquantization level is applied to the fourth portion 440, and a moderatequantization level is applied to the second portion 420 and the thirdportion 430.

FIG. 4C illustrates a diagram 460 showing relationships betweenquantization levels 470 and the activity variances of image portionsillustrated in FIG. 4A according to one embodiment. As shown in FIG. 4A,the relationship of between activity variances and quantization levelsamong the portions is: the quantization level for the first portion 410(with the highest activity level) is lower than the quantization levelfor the third portion 430 (with the second highest activity level),which is lower than the quantization level for the fourth portion 440(with the third highest activity level), which is lower than thequantization level for the second portion 420 (with the lowestactivity).

Image activity data can be used to determine a change in scene or toselect a GOP structure for a dynamic GOP. For example, a large change inactivity between two neighboring frames can be used as indication ofchange of scene. This change of scene information can be used fordynamically inserting I frame (or reference frame) or dynamicallychanging the GOP structure.

FIG. 5A illustrates the determination of a reference frame 511 of aseries of frames based on image activity data, in according to oneembodiment. A large change in scene (such as a black picture betweenframes scenes or the start of completely different shot) may show alarge activity variance per block. For instance, in a scene where a busmoves in a park, the activity variance between frames may be smallerthan in a scene where fireworks are exploding in a dark sky. Theactivity variance between frames can determine a scene change. In someembodiments, a histogram is calculated for activity variance betweenframes. For example, the activity variance between frames can becalculated as described above with regard to FIG. 3. This histogram isthen compared with a histogram calculated for a previous frame todetermine the activity variance between the two frames. In someembodiments, comparisons between frames may be a simple difference, ormay be more complex (e.g., by comparing the amplitude and frequency ofone or more factors between frames).

The activity variance between frames is compared against a thresholdvalue. If the activity variance between frames is greater than thethreshold value, a scene change can be detected. The threshold value maybe pre-determined, for instance based on known encoded content. Thisthreshold value can be tuned (or trained) based on the noise level andluminance level of the scene. As shown in FIG. 5A, a frame N-2 501 isencoded as P frame 511. Before encoding frame N-1 503A, block-basedactivity data of the unencoded frame N-1 503A is compared withblock-based activity data of a previously encoded frame N-2 501 tocalculate an activity variance between these two frames. The activityvariance between the frame N-1 503A and the frame N-2 501 is determinedto be below the threshold value. Accordingly, the frame N-1 503A isencoded as P frame 513. Before encoding frame N 505A, block-basedactivity data of the unencoded frame N 505A is compared with block-basedactivity data of a previously encoded frame N-1 503A to calculate anactivity variance between these two frames. The activity variancebetween the frame N 505A and the frame N-1 503A is determined to beabove the threshold value 509. Accordingly, a scene change is detected,and the frame N-1 503A is encoded as I frame 515.

In some embodiments, image activity data can be similarly applied to aGOP. For example, an activity variance between an unencoded image and anencoded image can be calculated. If the activity variance is above apredetermined threshold value, a scene change between the two images isdetected. The unencoded image can then be encoded as an I image.

In some embodiments, a large activity variance between neighboringframes for the same block indicates a sudden movement of object (orcamera) or appearance of new object in a scene captured in an image. Forexample, a ball suddenly appearing in a scene can be identified bycomparing activity variance with neighboring frames. In scenarios wherea new object appears in a scene for the first time, there is no previoushistory of the new object. Blocks associated with the new object can bemarked with higher weights or higher biases for Intra blocks. Whileencoding, the Intra/Inter decision could take these weights/biases inconsideration before classifying the block as an Intra- or Inter-typeblock. In other words, this algorithm can provide hints to the encoderso that the encoder can make a better block type decision.

FIG. 5B illustrates a block type determination based on image activitydata, according to one embodiment. An activity variance of each block islisted in a respective block location of the frame N-1 503B and theframe N 505B. A large difference 517 in activity variance of blocksbetween neighboring frames (e.g., the frame N-1 503B and the frame N505B) is determined and circled within FIG. 5B. These blocks are flaggedwith higher weights (or bias), and these weights/biases can be used toclassify blocks as Intra- or Inter-blocks. A larger difference inactivity variance among the same blocks in frames N-1 503B and N 505B isindicated by assigning a higher weight. In some embodiments, a higherweight indicates a higher probability of a block being an Inter block,whereas in other embodiments, a high weight indicates a higherprobability of a block being an Intra block. In some embodiments, asmaller difference in activity variance between neighboring framesindicates a lower probability of a block being an Intra block (sinceblock has smaller change/movement from previous frame and a history ismaintained), while in other embodiments, a smaller difference inactivity variances indicates a lower probability of a block being anInter block. It should be noted that in some embodiments, the likelihoodthat a block is either an Intra block or an Inter block is eitherdirectly or inversely proportional to an activity variance associatedwith the block.

In some embodiments, an activity variance of each block can be used todetermine block transform size and block prediction size. In cases wherebit preservation is needed in high activity areas, smaller transform andprediction (either Intra- or Inter-block) can be used. Based on activityvariance, an encoder can use the appropriate transform and predictionsizes. For example, an encoder can use activity data to select atransform and prediction block size of 4×4, 8×8, or 16×16 based on imageactivity data. This method can be used to assistcomputationally-intensive algorithms such as rate-distortion operationsto reduce computations and save power in real time during encoding.

FIG. 5C illustrates the determination of a transform size based on imageactivity data, in according to one embodiment. An activity variance ofeach block is listed in a respective block location of the frame N 520.The activity variance of each block is used to determine a respectivetransform block size for the block. As shown in FIG. 5C, a largertransform size (e.g., 16×16) can be applied to blocks with loweractivity variances (such as activity values that are below or equal to16). Likewise, a smaller transform size can be applied to blocks withhigher activity variances (such as activity variances that are above orequal to 200).

In some embodiments, a similar method based on image activity data canbe applied to prediction size. For example, a larger prediction size canbe applied to blocks with lower activity variances, while a smallerprediction size can be applied to blocks with higher activity variances.

In some embodiments, the reference frame or picture (e.g., I frame or Ipicture) can be determined based on image luminance data. For example, aluminance level of each block is calculated. If a difference inluminance level between an encoded frame and an unencoded frame is abovea predetermined threshold, a scene change is determined, and theunencoded frame can be encoded as I frame. This can be in addition toactivity variance based scene change detection to avoid any falseresults.

FIG. 6 illustrates an example high-level block diagram of an imageprocessing engine 116C, according to one embodiment. In the embodimentof FIG. 6, the image processing engine 116C includes an image pipeline620, a pre-encoding module 630, and an encoder 640. The statistics data660 includes image luminance data and image activity data. Both oreither of the image pipeline 620 and the pre-encoding module 630 cangenerate the statistics data 660 and store the statistics data 660 intothe memory 104. The pre-encoding module 630 generates the YUV data 670and stores the YUV data 670 into the memory 104. The encoder 640extracts the statistics data 660 and the YUV data 670 from the memory104, analyzes the data, and generates compressed encoded data based onthe data 660 and the YUV data 670. Data reduction operations can beperformed before storing data in the memory 104 (for instance, eachrange of luminance levels or activity data can be put in buckets tolimit the total number of levels and activity variances). In someembodiments, the encoder 640 determines a quantization level based onboth a luminance level and an activity variance of each block in the YUVdata. It should be noted that although embodiments described hereininclude the use of a memory to store data 660 prior to encoding by theencoder 640, in other embodiments, the encoder 640 receives thestatistics data directly from the image pipeline 620 and pre-encodingmodule 630 without the statistics data first being stored in the memory.

In one embodiment, the encoder 640 determines a quantization level foreach block based on both a corresponding luminance level and acorresponding activity variance. For example, the encoder 640 determinesa quantization level for a block based on a luminance level of theblock, and then adjusts the determined quantization level based on anactivity variance of the block such that the adjusted quantization levelis able to compress the block at an above-threshold level of compressionwhile preserving an above-threshold amount of image information, forinstance to preserve image information in portions of the image withboth high activity levels and low luminance levels. Similarly, theencoder 640 can determine a quantization level for a block based on anactivity variance of the block first, and then can adjust the determinedquantization level based on a luminance level of the block. In someembodiments, the encoder 640 weights quantization levels based on aluminance level and an activity variance of a block. In suchembodiments, if the luminance level is much greater than the activityvariance, the encoder 640 weights the quantization level based more onthe luminance level, and vice versa.

Encoding Image Data Based on Image Luminance Data

FIG. 7 illustrates a process 700 for encoding image data based on imageluminance data, according to one embodiment. In some embodiments, theprocess 700 is performed by the camera system 100. In some embodiments,the method includes different, additional, or fewer steps than thosedepicted by FIG. 7, and the steps described in conjunction with FIG. 7can be performed in different orders.

The camera system 100 converts 710 light incident upon an image sensorinto raw image data. For example, the image sensor 112 converts lightincident upon an image sensor into raw RGB image data. The camera system100 then converts 720 raw image data into color-space image data. Forexample, the image processing engine 116 coverts the raw RGB image datainto YUV data.

The camera system 100 calculates 730 luminance levels of the color-spaceimage data. For example, the image processing engine 116 identifies aplurality of blocks in the YUV data and calculates a luminance level foreach block. The image processing engine 116 may determine a luminancelevel for a block by using a sum of pixel values in the block.

The camera system 100 stores 740 the color-space image data and theluminance levels into the memory 104. The camera system 100 extracts 750the color-space image data and the luminance levels from the memory 104.It should be noted that in some embodiments, the storage 740 ofcolor-space image data into memory and the extraction 750 of thecolor-space image data from memory are bypassed.

The camera system 100 encodes 760 the color-space image data based onthe luminance levels. For example, the camera 100 determines aquantization level for each block based on a luminance levelcorresponding to the block. If a luminance level for a block indicatesthat the block is darker, the camera system 100 can select a lowerquantization level (e.g., a quantization level with less quantizationstrength) than for a block corresponding to a brighter luminance level.The encoded image data can then be stored in memory, can be processedusing one or more processing operations, or can be outputted to adisplay, for instance, for preview by a user of the camera.

Encoding Image Data Based on Image Activity Data

FIG. 8 illustrates a process 800 for encoding image data based on imageactivity data, according to one embodiment. In some embodiments, theprocess 800 is performed by the camera system 100. In some embodiments,the method includes different, additional, or fewer steps than thosedepicted by FIG. 8, and the steps described in conjunction with FIG. 8can be performed in different orders.

The camera system 100 converts 810 light incident upon an image sensorinto raw image data. For example, the image sensor 112 converts lightincident upon an image sensor into raw RGB image data. The camera system100 then converts 820 raw image data into color-space image data. Forexample, the image processing engine 116 coverts the raw RGB image datainto YUV data.

The camera system 100 calculates 830 activity variances of thecolor-space image data. For example, the image processing engine 116identifies a plurality of blocks in the YUV data and calculates anactivity variance for each block. The image processing engine 116 maydetermine an activity variance for a block by adding a sum ofdifferences of pixel values along a vertical direction of the YUV dataand a sum of differences of pixel values along a horizontal direction ofthe YUV data in the block.

The camera system 100 stores 840 the color-space image data and theactivity variances into the memory 104. The camera system 100 extracts850 the color-space image data and the activity variances from thememory 104. It should be noted that in some embodiments, the storage 840of color-space image data into memory and the extraction 850 of thecolor-space image data from memory are bypassed.

The camera system 100 encodes 860 the color-space image data based onthe activity variances. For example, the camera system 100 determines aquantization level for each block based on an activity variancecorresponding to the block. If an activity variance for a blockindicates that the block has a higher activity level, the camera system100 can select a lower quantization level than for a block correspondingto a lower activity level. The encoded image data can then be stored inmemory, can be processed using one or more processing operations, or canbe outputted to a display, for instance, for preview by a user of thecamera.

Additional Configuration Considerations

Certain embodiments are described herein as including logic or a numberof components, modules, or mechanisms, for example, as illustrated inFIGS. 2 and 3. Modules may constitute either software modules (e.g.,code embodied on a machine-readable medium or in a transmission signal)or hardware modules. A hardware module is tangible unit capable ofperforming certain operations and may be configured or arranged in acertain manner. In example embodiments, one or more computer systems(e.g., a standalone, client or server computer system) or one or morehardware modules of a computer system (e.g., a processor or a group ofprocessors) may be configured by software (e.g., an application orapplication portion) as a hardware module that operates to performcertain operations as described herein.

The performance of certain of the operations may be distributed amongthe one or more processors, not only residing within a single machine,but deployed across a number of machines. In some example embodiments,the one or more processors or processor-implemented modules may belocated in a single geographic location (e.g., within a homeenvironment, an office environment, or a server farm). In other exampleembodiments, the one or more processors or processor-implemented modulesmay be distributed across a number of geographic locations.

Unless specifically stated otherwise, discussions herein using wordssuch as “processing,” “computing,” “calculating,” “determining,”“presenting,” “displaying,” or the like may refer to actions orprocesses of a machine (e.g., a computer) that manipulates or transformsdata represented as physical (e.g., electronic, magnetic, or optical)quantities within one or more memories (e.g., volatile memory,non-volatile memory, or a combination thereof), registers, or othermachine components that receive, store, transmit, or displayinformation.

Some embodiments may be described using the expression “coupled” and“connected” along with their derivatives. For example, some embodimentsmay be described using the term “coupled” to indicate that two or moreelements are in direct physical or electrical contact. The term“coupled,” however, may also mean that two or more elements are not indirect contact with each other, but yet still co-operate or interactwith each other. The embodiments are not limited in this context.Further, unless expressly stated to the contrary, “or” refers to aninclusive or and not to an exclusive or.

Upon reading this disclosure, those of skill in the art will appreciatestill additional alternative structural and functional designs for asystem and a process for synchronizing multiple image sensors throughthe disclosed principles herein. Thus, while particular embodiments andapplications have been illustrated and described, it is to be understoodthat the disclosed embodiments are not limited to the preciseconstruction and components disclosed herein. Various apparentmodifications, changes and variations may be made in the arrangement,operation and details of the method and apparatus disclosed hereinwithout departing from the spirit and scope defined in the appendedclaims.

What is claimed is:
 1. A camera system, comprising: an image sensorconfigured to convert light incident upon the image sensor into rawimage data; an image pipeline configured to: convert the raw image datainto color-space image data; and calculate luminance levels of thecolor-space image data; an encoder configured to: determine quantizationlevels of the color-space image data based on the luminance levels; andencode the color-space image data using the determined quantizationlevels to produce encoded image data; and a memory configured to storethe encoded image data.
 2. The camera system of claim 1, whereincalculating luminance levels of the color-space image data comprises:identifying a plurality of blocks of the color-space image data; andcalculating a luminance level of each block.
 3. The camera system ofclaim 2, wherein calculating the luminance level of each blockcomprises: calculating a sum of pixel values in each block.
 4. Thecamera system of claim 2, wherein determining quantization levels of thecolor-space image data based on the luminance levels comprises:determining a quantization level for each block based on a correspondingluminance level of the block.
 5. The camera system of claim 4, wherein agreater quantization level is determined for a first block than for asecond block darker than the first block.
 6. The camera system of claim1, wherein the image pipeline is further configured to calculateactivity variances of the color-space image data, wherein the encoder isfurther configured to determine quantization levels of the color-spaceimage data based on the activity variances and to encode the color-spaceimage data using the determined quantization levels, and wherein thememory further is configured to store the color-space image data and theactivity variances.
 7. The camera system of claim 6, wherein at leastone quantization level is determined based on both an activity varianceand a luminance level.
 8. The camera system of claim 1, wherein theimage pipeline is further configured to store the color-space image dataand luminance levels into the memory, and the encoder is furtherconfigured to extract the color-space image data and luminance levelsfrom the memory.
 9. The camera system of claim 1, wherein the encoder isfurther configured to determine a reference frame and GOP structurebased on the luminance levels.
 10. The camera system of claim 1, furthercomprising a pre-encoding module configured to pre-process the encodedimage data using one or more pre-processing operations.
 11. A method,comprising: capturing, by a camera, light incident upon an image sensorto produce raw image data; converting, by the camera, the raw image datainto color-space image data; calculating, by the camera, luminancelevels of the color-space image data; determining, by the camera,quantization levels of the color-space image data based on the luminancelevels; and encoding, by the camera, the color-space image data usingthe determined quantization levels to produce encoded image data. 12.The method of claim 11, wherein calculating luminance levels of thecolor-space image data comprises: identifying a plurality of blocks ofthe color-space image data; and calculating a luminance level of eachblock.
 13. The method of claim 12, wherein calculating the luminancelevel of each block comprises: calculating a sum of pixel values in eachblock.
 14. The method of claim 12, wherein determining quantizationlevels of the color-space image data based on the luminance levelscomprises: determining a quantization level for each block based on acorresponding luminance level of the block.
 15. The method of claim 14,wherein a greater quantization level is determined for a first blockthan for a second block darker than the first block.
 16. The method ofclaim 11, further comprising: calculating activity variances of thecolor-space image data; determining quantization levels of thecolor-space image data further based on the activity variances; encodingthe color-space image data using the determined quantization levels; andstoring the color-space image data and the activity variances.
 17. Themethod of claim 16, wherein at least one quantization level isdetermined based on both an activity variance and a luminance level. 18.The method of claim 11, further comprising storing the color-space imagedata and luminance levels into a memory, wherein the color-space imagedata and luminance levels are extracted from the memory before thequantization levels are determined.
 19. The method of claim 11, furthercomprising determining a reference frame and GOP structure based on theluminance levels.
 20. The method of claim 11, further comprisingpre-processing the encoded image data using one or more pre-processingoperations.